WIP: Validate HAProxy 2.8.27 and 3.2.22 - #832
Conversation
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Warning Review limit reached
Next review available in: 26 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe HAProxy image Dockerfiles now install pinned HAProxy RPMs from remote URLs instead of installing HAProxy through 🚥 Pre-merge checks | ✅ 14 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@images/router/haproxy/Dockerfile.ocp`:
- Around line 3-9: Combine the HAProxy RPM installation and package-cache
cleanup into the same RUN layer, preserving the existing haproxy verification
and package installation behavior. Apply this change in
images/router/haproxy/Dockerfile.ocp lines 3-9,
images/router/haproxy28/Dockerfile.ocp lines 3-9, and
images/router/haproxy32/Dockerfile.ocp lines 3-9 so no yum cache remains in the
final image layer.
- Line 3: Replace the mutable GitHub branch-based RPM URLs in
images/router/haproxy/Dockerfile.ocp:3-3,
images/router/haproxy28/Dockerfile.ocp:3-3, and
images/router/haproxy32/Dockerfile.ocp:3-3 with immutable release or commit
references, and verify each downloaded RPM using its expected checksum or a
trusted signature before installation.
- Line 4: Replace the version-agnostic HAProxy checks at
images/router/haproxy/Dockerfile.ocp:4,
images/router/haproxy28/Dockerfile.ocp:4, and
images/router/haproxy32/Dockerfile.ocp:4 with assertions that verify the
installed HAProxy binary reports exactly 3.2.22, 2.8.27, and 3.2.22
respectively, while failing the build on any mismatch.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 169e650f-2885-476c-96d7-326c4da269c3
📒 Files selected for processing (3)
images/router/haproxy/Dockerfile.ocpimages/router/haproxy28/Dockerfile.ocpimages/router/haproxy32/Dockerfile.ocp
| FROM registry.ci.openshift.org/ocp/5.0:haproxy-router-base | ||
| RUN INSTALL_PKGS="socat haproxy32 rsyslog procps-ng util-linux" && \ | ||
|
|
||
| RUN yum install -y https://github.com/jcmoraisjr/openshift-router/raw/refs/heads/rpms/haproxy32-3.2.22-1.rhocp.el9.x86_64.rpm |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Files matching Dockerfile.ocp:\n'
git ls-files | grep -E '(^|/)Dockerfile\.ocp$' | sort
printf '\nRelevant snippets:\n'
for f in images/router/haproxy/Dockerfile.ocp images/router/haproxy2十八条/Dockerfile.ocp images/router/haproxy32/Dockerfile.ocp; do
if [ -f "$f" ]; then
echo "--- $f"
sed -n '1,40p' "$f" | cat -n
fi
done
printf '\nSearch router Dockerfiles for yum/rpm installation and references:\n'
rg -n "yum|dnf|rpm|sha|rhaos|rhocp|openshift-router|github.com/jcmoraisjr" images/router -g 'Dockerfile.ocp' || trueRepository: openshift/router
Length of output: 4373
Pin the RPM content by digest or checksum.
All three images/router/*haproxy*/Dockerfile.ocp files install RPMs from github.com/jcmoraisjr/openshift-router/raw/refs/heads/rpms/. That branch URL can change the package contents without an unchanged Dockerfile being updated. Use an immutable release/commit reference and verify the RPM hash or trusted signature at each install site.
🧰 Tools
🪛 Trivy (0.72.0)
[error] 3-3: 'yum clean all' missing
'yum clean all' is missed: yum install -y https://github.com/jcmoraisjr/openshift-router/raw/refs/heads/rpms/haproxy32-3.2.22-1.rhocp.el9.x86_64.rpm
Rule: DS-0015
(IaC/Dockerfile)
📍 Affects 3 files
images/router/haproxy/Dockerfile.ocp#L3-L3(this comment)images/router/haproxy28/Dockerfile.ocp#L3-L3images/router/haproxy32/Dockerfile.ocp#L3-L3
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@images/router/haproxy/Dockerfile.ocp` at line 3, Replace the mutable GitHub
branch-based RPM URLs in images/router/haproxy/Dockerfile.ocp:3-3,
images/router/haproxy28/Dockerfile.ocp:3-3, and
images/router/haproxy32/Dockerfile.ocp:3-3 with immutable release or commit
references, and verify each downloaded RPM using its expected checksum or a
trusted signature before installation.
| RUN yum install -y https://github.com/jcmoraisjr/openshift-router/raw/refs/heads/rpms/haproxy32-3.2.22-1.rhocp.el9.x86_64.rpm | ||
| RUN haproxy -vv | ||
|
|
||
| RUN INSTALL_PKGS="socat rsyslog procps-ng util-linux" && \ | ||
| yum install -y --setopt=install_weak_deps=0 $INSTALL_PKGS && \ | ||
| rpm -V $INSTALL_PKGS && \ | ||
| yum clean all && \ |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win
Clean package caches in the same installation layer.
The HAProxy RPM installation runs before the later yum clean all, so the earlier layer's downloaded bytes remain in the image.
images/router/haproxy/Dockerfile.ocp#L3-L9: combine the HAProxy RPM installation and cleanup in oneRUN.images/router/haproxy28/Dockerfile.ocp#L3-L9: combine the HAProxy RPM installation and cleanup in oneRUN.images/router/haproxy32/Dockerfile.ocp#L3-L9: combine the HAProxy RPM installation and cleanup in oneRUN.
Trivy reports this missing cleanup. As per path instructions, container builds must have “No package manager cache in final layer”.
🧰 Tools
🪛 Trivy (0.72.0)
[error] 3-3: 'yum clean all' missing
'yum clean all' is missed: yum install -y https://github.com/jcmoraisjr/openshift-router/raw/refs/heads/rpms/haproxy32-3.2.22-1.rhocp.el9.x86_64.rpm
Rule: DS-0015
(IaC/Dockerfile)
📍 Affects 3 files
images/router/haproxy/Dockerfile.ocp#L3-L9(this comment)images/router/haproxy28/Dockerfile.ocp#L3-L9images/router/haproxy32/Dockerfile.ocp#L3-L9
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@images/router/haproxy/Dockerfile.ocp` around lines 3 - 9, Combine the HAProxy
RPM installation and package-cache cleanup into the same RUN layer, preserving
the existing haproxy verification and package installation behavior. Apply this
change in images/router/haproxy/Dockerfile.ocp lines 3-9,
images/router/haproxy28/Dockerfile.ocp lines 3-9, and
images/router/haproxy32/Dockerfile.ocp lines 3-9 so no yum cache remains in the
final image layer.
Sources: Path instructions, Linters/SAST tools
0eab346 to
2a91b85
Compare
|
/test ? |
|
Running all haproxy28 related tests: /test e2e-agnostic-haproxy28-techpreview |
|
Provision failed:
/test e2e-agnostic |
|
/testwith openshift/router/master/e2e-agnositc-haproxy28-techpreview openshift/origin#31494 |
|
@gcs278, |
|
/testwith openshift/router/master/e2e-agnostic-haproxy28-techpreview openshift/origin#31494 |
|
/testwith openshift/router/master/e2e-aws-serial-haproxy28-techpreview-1of2 openshift/origin#31494 |
|
@gcs278, |
|
/testwith openshift/router/master/e2e-aws-serial-haproxy28-techpreview-2of2 openshift/origin#31494 |
|
@gcs278, |
|
/testwith openshift/router/master/e2e-upgrade-haproxy28-techpreview openshift/origin#31494 |
|
/testwith openshift/router/master/e2e-aws-serial-haproxy28-techpreview openshift/origin#31494 |
|
@gcs278, |
|
@gcs278, |
|
/testwith openshift/router/master/e2e-agnostic-haproxy28-techpreview openshift/origin#31494 |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
base_url='https://prow.ci.openshift.org/view/gs/test-platform-results/logs/multi-pr-openshift-router-832-openshift-origin-31494-e2e-agnostic-haproxy28-techpreview/2085712441596973056'
artifact_prefix='https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/multi-pr-openshift-router-832-openshift-origin-31494-e2e-agnostic-haproxy28-techpreview/2085712441596973056'
echo '== Prow page =='
curl --fail --location --silent --show-error "$base_url" | sed -n '1,100p'
for file in prowjob.json started.json finished.json build-log.txt; do
echo
echo "== ${file} =="
curl --fail --location --silent --show-error "${artifact_prefix}/${file}" \
| tail -n 300 \
|| true
doneLength of output: 44817 🏁 Script executed: #!/bin/bash
set -euo pipefail
prefix='https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/multi-pr-openshift-router-832-openshift-origin-31494-e2e-agnostic-haproxy28-techpreview/2085712441596973056'
log_file="$(mktemp)"
trap 'rm -f "$log_file"' EXIT
curl --fail --location --silent --show-error "${prefix}/build-log.txt" >"$log_file"
echo '== nodes-readiness and timeout messages =='
grep -inE -C 12 'nodes-readiness|ready nodes expected|timeout reached|not ready|ContainerFailed|failed after 30m' "$log_file" || true
echo
echo '== readiness step metadata =='
curl --fail --location --silent --show-error \
"${prefix}/artifacts/e2e-agnostic-haproxy28-techpreview/nodes-readiness/finished.json" \
|| true
echo
echo '== artifact references near nodes-readiness =='
grep -inE -C 3 'artifacts/.{0,100}(nodes-readiness|nodes)' "$log_file" || trueLength of output: 13215
The The incomplete MachineSet was: The other two worker MachineSets were Ready. The cluster therefore had three control-plane nodes and only two Ready worker nodes. This failure is in CI infrastructure or cluster provisioning. The log does not show an HAProxy error, a router test failure, or evidence that either PR caused the failure. Retrying the job is appropriate. You are interacting with an AI system. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
jobs=(
'multi-pr-openshift-router-832-openshift-origin-31494-e2e-agnostic-haproxy28-techpreview/2085760173623742464'
'multi-pr-openshift-router-832-openshift-origin-31494-e2e-aws-serial-haproxy28-techpreview/2085718225097592832'
)
prefix='https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs'
for job in "${jobs[@]}"; do
echo "===== JOB: $job ====="
log="$(mktemp)"
trap 'rm -f "$log"' EXIT
curl --fail --location --silent --show-error "$prefix/$job/build-log.txt" > "$log"
echo '--- finished.json ---'
curl --fail --location --silent --show-error "$prefix/$job/finished.json" || true
echo
echo '--- failure signatures and test summary ---'
grep -inE -C 10 \
'fail(ure|ed)?|error:|panic:|timed out|timeout|not ready|test .*(failed|passed)|[0-9]+ specs?, [0-9]+ failures?|ginkgo' \
"$log" | tail -n 500 || true
echo '--- final log section ---'
tail -n 180 "$log"
rm -f "$log"
trap - EXIT
echo
doneLength of output: 201535 🏁 Script executed: #!/bin/bash
set -euo pipefail
bucket='test-platform-results'
jobs=(
'logs/multi-pr-openshift-router-832-openshift-origin-31494-e2e-agnostic-haproxy28-techpreview/2085760173623742464'
'logs/multi-pr-openshift-router-832-openshift-origin-31494-e2e-aws-serial-haproxy28-techpreview/2085718225097592832'
)
for job in "${jobs[@]}"; do
echo "===== $job ====="
prefix="${job}/artifacts/junit/"
objects="$(
curl --fail --location --silent --show-error \
--get "https://storage.googleapis.com/storage/v1/b/${bucket}/o" \
--data-urlencode "prefix=${prefix}" \
--data-urlencode 'fields=items(name),nextPageToken' \
| jq -r '.items[]?.name'
)"
printf '%s\n' "$objects" | grep -E 'junit_e2e.*\.xml$|junit.*\.xml$' | head -n 30
while IFS= read -r object; do
[ -n "$object" ] || continue
case "$object" in
*junit_e2e*.xml)
echo "--- $object: failed test cases ---"
curl --fail --location --silent --show-error \
"https://storage.googleapis.com/download/storage/v1/b/${bucket}/o/$(jq -rn --arg v "$object" '$v|`@uri`')?alt=media" \
| python3 -c '
import sys
import xml.etree.ElementTree as ET
root = ET.parse(sys.stdin).getroot()
for case in root.iter("testcase"):
failure = case.find("failure")
error = case.find("error")
if failure is not None or error is not None:
node = failure if failure is not None else error
print(f"classname={case.get("classname")}\nname={case.get("name")}\ntype={node.get("type")}\nmessage={node.get("message")}\noutput={(node.text or "").strip()[:5000]}\n---")
'
;;
esac
done <<< "$objects"
doneLength of output: 2076 🏁 Script executed: #!/bin/bash
set -euo pipefail
base='https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs'
reports=(
'multi-pr-openshift-router-832-openshift-origin-31494-e2e-agnostic-haproxy28-techpreview/2085760173623742464/artifacts/e2e-agnostic-haproxy28-techpreview/junit/junit_e2e__20260807-171953.xml'
)
for report in "${reports[@]}"; do
echo "===== $report ====="
curl --fail --location --silent --show-error "$base/$report" \
| python3 -c '
import sys
import xml.etree.ElementTree as ET
root = ET.parse(sys.stdin).getroot()
found = False
for case in root.iter("testcase"):
node = case.find("failure") or case.find("error")
if node is not None:
found = True
print("classname:", case.get("classname"))
print("name:", case.get("name"))
print("message:", node.get("message"))
print((node.text or "").strip()[:8000])
print("---")
if not found:
print("No failed test cases in this report.")
'
doneLength of output: 1861
|
|
/testwith openshift/router/master/e2e-aws-serial-haproxy28-techpreview openshift/origin#31494 |
|
/testwith openshift/router/master/e2e-agnostic-haproxy28-techpreview openshift/origin#31494 |
|
/test e2e-metal-ipi-ovn-dualstack |
|
/payload-job periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-upgrade |
|
@gcs278: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/399a4c60-94de-11f1-895c-5343e08ac569-0 |
|
/payload-aggregate periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn 5 |
|
@gcs278: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/7dec1330-94de-11f1-93c6-3820c168e1f9-0 |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
payload_urls=(
'https://pr-payload-tests.ci.openshift.org/runs/ci/399a4c60-94de-11f1-895c-5343e08ac569-0'
'https://pr-payload-tests.ci.openshift.org/runs/ci/7dec1330-94de-11f1-93c6-3820c168e1f9-0'
)
prow_page='https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_router/832/pull-ci-openshift-router-master-e2e-metal-ipi-ovn-dualstack/2086837104960081920'
prow_prefix='https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_router/832/pull-ci-openshift-router-master-e2e-metal-ipi-ovn-dualstack/2086837104960081920'
for url in "${payload_urls[@]}"; do
echo "===== PAYLOAD RUN: $url ====="
curl --fail --location --silent --show-error "$url" | sed -n '1,240p' || true
echo
done
echo "===== PROW PAGE ====="
curl --fail --location --silent --show-error "$prow_page" | sed -n '1,180p' || true
echo "===== PROW METADATA ====="
for object in prowjob.json started.json finished.json; do
echo "--- $object ---"
curl --fail --location --silent --show-error "$prow_prefix/$object" || true
echo
done
echo "===== PROW FAILURE SIGNALS ====="
log="$(mktemp)"
trap 'rm -f "$log"' EXIT
curl --fail --location --silent --show-error "$prow_prefix/build-log.txt" >"$log" || true
grep -inE -C 6 \
'haproxy|router|ingress|route|failure|failed|error:|panic:|timeout|[0-9]+ specs?, [0-9]+ failures?' \
"$log" | tail -n 1200 || true
echo "===== PROW ARTIFACT INDEX ====="
curl --fail --location --silent --show-error \
'https://storage.googleapis.com/storage/v1/b/test-platform-results/o?prefix=pr-logs%2Fpull%2Fopenshift_router%2F832%2Fpull-ci-openshift-router-master-e2e-metal-ipi-ovn-dualstack%2F2086837104960081920%2Fartifacts%2F&fields=items(name),nextPageToken' \
| jq -r '.items[]?.name' | head -n 400 || trueLength of output: 202362 🏁 Script executed: #!/bin/bash
set -euo pipefail
prow_prefix='https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_router/832/pull-ci-openshift-router-master-e2e-metal-ipi-ovn-dualstack/2086837104960081920'
junit_path='artifacts/e2e-metal-ipi-ovn-dualstack/baremetalds-e2e-test/artifacts/junit/junit_e2e__20260810-170905.xml'
echo '===== PROW FAILED JUNIT CASES ====='
curl --fail --location --silent --show-error "$prow_prefix/$junit_path" |
python3 -c '
import sys
import xml.etree.ElementTree as ET
root=ET.parse(sys.stdin).getroot()
for tc in root.iter("testcase"):
node=tc.find("failure")
if node is None:
node=tc.find("error")
if node is not None:
print("CLASS:",tc.get("classname"))
print("NAME:",tc.get("name"))
print("TYPE:",node.get("type"))
print("MESSAGE:",node.get("message"))
print((node.text or "").strip()[:16000])
print("---")
'
echo '===== HAProxy IMAGE BUILD VERIFICATION ====='
for image in haproxy-router-amd64 haproxy-router-haproxy28-amd64 haproxy-router-haproxy32-amd64; do
echo "--- $image ---"
curl --fail --location --silent --show-error "$prow_prefix/artifacts/build-logs/$image.log" |
grep -inE -C 3 'haproxy.*(version|2\.8\.27|3\.2\.22)|error|failed|complete|success' | tail -n 160 || true
done
for id in '399a4c60-94de-11f1-895c-5343e08ac569-0' '7dec1330-94de-11f1-93c6-3820c168e1f9-0'; do
page="$(mktemp)"
trap 'rm -f "$page"' EXIT
curl --fail --location --silent --show-error \
"https://pr-payload-tests.ci.openshift.org/runs/ci/$id" >"$page"
echo "===== PAYLOAD $id: PAGE SIGNALS ====="
grep -inE -C 3 \
'failed|failure|error|success|passed|job|prow|test-platform-results|api/' \
"$page" | head -n 500 || true
echo "===== PAYLOAD $id: LINK TARGETS ====="
grep -oE 'https?://[^"[:space:]<>]+' "$page" |
sed 's/[),;]$//' |
grep -Ei 'prow|test-platform|payload|ci-openshift|gcs' |
sort -u | head -n 200 || true
rm -f "$page"
trap - EXIT
doneLength of output: 28704 🏁 Script executed: #!/bin/bash
set -euo pipefail
jobs=(
'logs/openshift-router-832-ci-5.0-e2e-aws-ovn-upgrade/2086862478704119808'
'logs/aggregator-periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn/2086862963305615360'
)
gcs='https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results'
for job in "${jobs[@]}"; do
prefix="$gcs/$job"
echo "===== $job: METADATA ====="
for obj in prowjob.json started.json finished.json; do
echo "--- $obj ---"
curl --fail --location --silent --show-error "$prefix/$obj" || true
echo
done
echo "===== $job: BUILD LOG SUMMARY ====="
curl --fail --location --silent --show-error "$prefix/build-log.txt" |
grep -inE -C 5 \
'blocking test failures|informing test failures|suite run returned|[0-9]+ specs?, [0-9]+ failures?|process timed out|failed|error:|haproxy|router|ingress|route' |
tail -n 800 || true
echo "===== $job: ARTIFACT NAMES ====="
encoded_prefix="$(printf '%s/artifacts/' "$job" | jq -sRr `@uri`)"
curl --fail --location --silent --show-error \
"https://storage.googleapis.com/storage/v1/b/test-platform-results/o?prefix=$encoded_prefix&fields=items(name),nextPageToken" |
jq -r '.items[]?.name' |
grep -E 'junit.*\.(xml|json)$|e2e.*\.log$|build-log\.txt$|finished\.json$' |
head -n 300 || true
done
bare_prefix="$gcs/pr-logs/pull/openshift_router/832/pull-ci-openshift-router-master-e2e-metal-ipi-ovn-dualstack/2086837104960081920"
e2e_path='artifacts/e2e-metal-ipi-ovn-dualstack/baremetalds-e2e-test/artifacts/e2e.log'
log="$(mktemp)"
trap 'rm -f "$log"' EXIT
curl --fail --location --silent --show-error "$bare_prefix/$e2e_path" >"$log"
echo '===== BARE-METAL: CONFLICTING STATUS TEST CONTEXT ====='
grep -inE -C 35 \
'The HAProxy router converges when multiple routers are writing conflicting status|stress\.go:240|conflicting status' \
"$log" || true
echo '===== BARE-METAL: HAProxy OR ROUTER ERROR SIGNALS ====='
grep -inE -C 4 \
'router-e2e|haproxy.*(error|fatal|panic|fail)|error.*haproxy|segmentation fault|back-end|backend.*down|connection refused' \
"$log" | tail -n 700 || trueLength of output: 103868
The failed test routers reached These occurred before the first successful reload and readiness checks. The test log does not show an HAProxy crash, configuration parse error, reload failure, or failed route traffic. Several nearby router tests passed. These include normal multi-router status convergence, overlapping route configurations, router health checks, route metrics, and unrecognized-host handling. Conclusion: The bare-metal result is worth retrying because it is a router status test. I would not block the HAProxy update on this single timeout unless it reproduces. The successful AWS upgrade run is positive evidence for the candidate images. You are interacting with an AI system. |
|
Perf & Scale results of haproxy 3.2.19 vs 3.2.22 HAProxy 3.2.22 vs 3.2.19 perf results look clean. No major concerns. RPS (requests/sec):
Latency:
HAProxy CPU:
CPU is a few percent higher on edge/http but lower on reencrypt and flat on passthrough — no consistent directional trend, looks like run-to-run variance on shared cloud infra rather than a real regression. Zero errors/timeouts on both. |
|
/test ? |
|
/test e2e-agnostic-haproxy28 |
|
@jcmoraisjr: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
HAProxy 2.8.27 vs 2.8.18 perf results look acceptable. No major concerns. Note: There's a bug with the Perf & Scale logic cloud-bulldozer/ingress-perf#88. It's reporting HAProxy 3.2 because it queries the monolithic router container. This run is actually HAProxy 2.8.27: Also, I'm comparing to 2.8 from July 8 (since that's the last time we had a periodic running with HAProxy 28 was the least noisy). We should also add periodic perf & scale for HAProxy28 versions in openshift/release#83066. HAProxy 2.8.27 vs 2.8.18 perf results (baseline: RPS (requests/sec):
Latency:
HAProxy CPU:
Edge and reencrypt show consistent RPS dips (~9% and ~5%) across all three baselines tested. Http and passthrough are flat or better. The pattern correlates with TLS termination — both affected termination types terminate client TLS, while the unaffected ones don't. HAProxy CPU is roughly flat, suggesting slightly less efficient per-request processing rather than doing dramatically more work. Zero errors/timeouts on both. Not terribly concerning - but the edge drop is something to note. |
|
/test perfscale-aws-ingress-perf-haproxy28 |
|
@jcmoraisjr I compared the most recent run too. HAProxy 2.8.27 vs 2.8.18 perf results (baseline: RPS (requests/sec):
Latency:
HAProxy CPU:
Edge dip is now ~13% on this run — worse than the previous ~9%. Reencrypt is ~8% worse. Http is slightly worse at ~2% (first time it's dipped). Passthrough still flat. The TLS termination correlation holds. However, let me run a new haproxy 2.8.18 - these are from last month, and I wonder if for some reason the Perf environment is slower now. I'll have to open a blank PR and run the test. |
|
Kicked off a 2.8.18 in #833 that we can make sure we compare with a current baseline |
|
/test perfscale-aws-ingress-perf-haproxy28 |
|
@jcmoraisjr another comparison, with a new baseline ran from #833. I think this mostly confirms: there really seems to be a slightly performance regression in edge: HAProxy 2.8.27 ( RPS (requests/sec):
Latency:
HAProxy CPU:
Edge dip (~9%) is consistent across multiple comparisons. Http, passthrough, and reencrypt are flat or better. Zero errors/timeouts on both. |
|
haproxy 2.8.27 updated to scratch-build in the same URL. /test perfscale-aws-ingress-perf-haproxy28 |
|
@jcmoraisjr with the new perf scale results. Actually looks even worse 😭 HAProxy 2.8.27 ( RPS (requests/sec):
Latency:
HAProxy CPU:
This is the worst run yet — edge dropped ~19% and reencrypt ~13%. All four termination types are worse. The 2.8.27 run ( |
|
/test perfscale-aws-ingress-perf-haproxy28 |
|
Patched 2.8.27, reverting /test perfscale-aws-ingress-perf-haproxy28 |
|
HAProxy 2.8.27 ( RPS (requests/sec):
Latency:
HAProxy CPU:
|
|
Running patched 2.8.27 once more. /test perfscale-aws-ingress-perf-haproxy28 |
|
HAProxy 2.8.27 ( They are mostly flat compared to 2.8.18 except the reencrypt test. Rencrypt however has also a much better CPU usage, decreasing in a similar rate, which suggests that the machine didn't provide everything HAProxy asked, maybe due to a noisy neighborhood during this test. Lets run another one as the tie break. RPS (requests/sec):
Latency:
HAProxy CPU:
|
|
Tie break /test perfscale-aws-ingress-perf-haproxy28 |
|
HAProxy 2.8.27 ( They are mostly flat now compared with 2.8.18, so we can confirm the previous reencrypt results was unrealistic. RPS (requests/sec):
Latency:
HAProxy CPU:
|
|
We're done, this can be closed now /close |
|
@jcmoraisjr: Closed this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
2 participants
No description provided.